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Abstract 


The Council for the Accreditation of Educator Preparation Programs (CAEP), required 
evidence of reliability and validity of measures used in a university's Educator 
Preparation Program (EPP). This paper describes processes that provided this evidence 
for the Teacher Performance Assessment (TPA). Literature examined included Messick 
(1989), Linn (1980). The TPA, a state-wide requirement, was introduced early in the 
university's EPP. Components were taught throughout coursework and clinical 
practice. Rubrics were aligned with Interstate Teacher Assessment and Support 
Consortium (InTASC) Model Core Teaching Standards and Learning Progressions for 
Teachers (CCSSO, 2017). Candidates were introduced to the TPA and rubrics in the 
Teacher Education Handbook. Course assignments were aligned with the TPA. 
Assessed knowledge and skills were taught for mastery. Accommodations were offered 
for candidates with documented learning differences. Candidates were given the 
opportunity throughout their work on the TPA to receive feedback and to revise 
sections. Three cycles of data were: Fall 2016: N=56; Spring 2017: N=94; and Fall 
2017: N=56. TPAs for Cycles | and 2 were identical. The TPA for Cycle 3 had been 
revised. The EPP consistently interpreted data from the four-point evaluation scale as 
interval-level. Construct and content validity and reliability evidence: Factor analysis 
results: Cycle I: seven subscales explained 76.76% of the variance. Cycle 2 seven 
subscales explained 73.9% of the variance. Cronbach's alpha reliabilities for Cycle I 
and II subscales ranged from a high of Contextual Factors: a=.907; a=.921 to a low of 
Analysis of Student Learning o=.882; o=.897. Reliabilities for Cycle II were High: 
Reflection and Self-Evaluation a= .881.and low Contextual Factors o=.673. Inter-rater 


reliabilities, consistently high, were included and discussed in the study. 
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The Validity and Reliability of the Teacher Performance Assessment (TPA) 
The Teacher Performance Assessment was selected for analyses of validity and 
reliability of measures because it is a comprehensive assessment that is introduced 
early in the program, is assessed through coursework, and is completed and evaluated 
during clinical practice late in the program. The rubrics for the TPA are aligned with 
InTASC Model Core Teaching Standards and Learning Progressions for Teachers 
(CCSSO, 2017). The performances are aligned with candidate requirements during 
clinical practice and are also aligned with the Kentucky Teacher Intern Performance 
(KTIP), the required in-field evaluation of first-year teachers in Kentucky. 
Validity 
Validity was defined by CAEP Glossary (2015) as: 
Validity. The extent to which a set of operations, test, or other 
assessment measures what it is supposed to measure. Validity is not a 
property of a data set but refers to the appropriateness of inferences 
from test scores or other forms of assessment and the credibility of the 
interpretations that are made concerning the findings of a measurement 
effort. 
The following aspects of validity described by Messick (1995) were examined: 
e Content-related validity: Do assessment items/components adequately 
and representatively sample the content area(s) to be measured? 
e Construct validity: Do assessments and the assessment system measure 
the content they purport to measure? 
e Fairness: Are all candidates afforded a fair opportunity to demonstrate 
their skills, knowledge, and dispositions? 
e Utility: How useful are the data generated from assessments? 
e Prediction (Criterion-related validity): How well do assessment 


instrument predict how well candidates will do in future situations? 
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e Consequences: Are assessment uses and interpretations contributing to 
increased candidate achievement and not producing unintended negative 
consequences? (Linn, 1994) 

Content and Construct Validity Evidence 

The content of the Techer Performance Assessment (TPA) is presented in the 
Teacher Education Program (TEP) Handbook online. All clinical practice students are 
required to complete a Teacher Performance Assessment (TPA). Compiling a TPA will 
provide you with professional growth experiences and documentation that reflects your 
ability to impact student learning. Working through the TPA design will assist you in 
developing “a teacher’s way of thinking.” This experience will prepare you for the 
Kentucky Teacher Internship Program (KTIP), which must be completed during your 
first year of teaching. (MSUTEPH, 2018). 

The TPA was designed to measure seven skills. The means for the measures 
across three cycles are presented below. The number of items per scale varied across 
items. The scoring rubric contained four score points. The mean values of the scales 
were all between the scale values were | (lowest value) to 4 (highest value). The mean 
values on the scales were between the 3 and 4 score point (Table 1). 


Table 1. Scale Means by Content Areas in Three Cycles of TPA Data 


Teacher Performance Assessed Fall 2016 Spring 2017 | Fall 2017* 
N=56 N=94 N=56 

Contextual Factors 3.24 3.42 3.67 
Learning Objectives 3.41 3.68 3.42 
Assessment Plan 3.28 3.44 3.21 
Design for Instruction eres 3.55 3.47 
Instructional Decision Making a2 3.48 Omitted 
Analysis of Student Learning 3.24 3.48 3.41 
Reflection and Self-Evaluation 3.14 3.47 3.28 
Formatting Expectations 3.74 3.63 One Item: 


*Fall 2017 Revised TPA 
Use of Factor Analyses to Examine Structure of the TPA by Cycle 
Factor analysis is a statistical technique that identifies the smaller number of 
factors/constructs/dimensions that underlie a larger set of variables (most of which are 
correlated to each other) (Sax, 1997). Factor analysis was used to examine the TPA 


data for each Cycle using SPSS software. Seven subscales were identified in Cycles | 
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and 2 data; six subscales were identified in Cycle 3 data. Cronbach's alpha internal- 
consistency reliabilities were computed for each subscale. Factor analysis was used to 
identify the number of components in the TPA. 

The structure of the TPA belongs to the data that are collected each cycle and 
not to the TPA instrument. Reliability is never an attribute of a test. Thompson and 
Vacha-Haase (2000) clearly explain this principle in their article "Psychometrics is 
Data Metrics: The test is not reliable." (Thompson & Vacha-Haase, 2000). Each time 
an instrument is used to collect data for measurement, reliability coefficients must be 
computed with the data collected. 


Table 2. Factor Analysis for Fall 2016 TPA Cycle I Data (N=56) 
Initial Eigenvalues Extraction Sums of Squared 
Loadings 
% of Cumulative % of Cumulative 


18.66 


1 
2 
3 
4 
2) 


7 
P8785 | 2.243 | 79.002 | 
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Seven components were extracted explaining 76.759% of the variance in the 
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a ee 
Le 
p 4 | 588 | 4536 | 68.944 | 1588 | 4536 | 68.944 
E=5e=] 
=a 
ees 


Fall 2016 data. The analysis confirmed the seven designed scales. This presents 
statistical evidence of construct validity and content validity of the Fall 2016 Teacher 
Performance Assessment data. The TPA measured what it was designed to measure. 
The reader is reminded that "structure" and reliability do not belong to an instrument. 
Instrument structure and reliabilities belong to the data that were collected. 

Cronbach alpha reliabilities are measures of internal consistency (Sax, 1997). 
Reliability coefficients do not "belong to a test," they belong to the data that were 
collected by an instrument. Thompson and Vacha-Hasse (2000) explained in "The Test 
is Not Reliable." 

Cronbach alpha reliabilities are frequently used as evidence of both consistence in 


measurement (the usual definition for reliability) but also for construct and content 
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validity. With high internal consistency reliability, users are assured of both 
credible measurement and strong internal reliability. This is excellent confirmation 
of the underlying construct measuring the content. 


Table 3. Factor Analysis for the Spring 2017 TPA Cycle 2 Data (N= 94) 


Initial Eigenvalues Extraction Sums of Squared 
Loadings 


% of Cumulative Total % of 
Variance % Variance 


Through factor analysis, seven components were identified in the Spring 2017 
TPA Data. The seven components explained 73.9% of the variance in the data (Table 3 
above). This presents statistical evidence of construct validity and content validity 
using the Spring 2017 Teacher Performance Assessment data. The TPA measured what 
it was designed to measure. 


Table 4. Factor Analysis for the Fall 2017 TPA Data: N=56 (SPSS) 


Initial Eigenvalues Extraction Sums of Squared 
Component : 
Loadings 
Total % of Cumulative | Total % of Cumulative 
Variance % Variance % 


| 6 | 17 | 4.007_—| 69.866 | 617 _| 4.007 _—|—69.866_| 
a 2 
| 8 434 | 2.820 | 76.198 | | 
a ae a a 
| tO 347 | 2.252 | 81208 || 


The Teacher Performance Assessment was revised prior to the Fall 2017 
semester. The purpose given for the revision was to clarify and simplify the rubrics. 


The Instructional Decision-Making scale was omitted in the Fall 2017 TPA. 
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Only one item was used to replace "Format Expectations" in the first two 
cycles. This one item was an evaluation of References. This one item did not constitute 
a "scale". In the analysis of the data, six scales were identified through factor analysis 
with SPSS. These six scales accounted for 69.86% of the variance, a decrease of 
variance explained from Fall 2016 (76.75%) and Spring 2017 (73.89%). 


Table 5. TPA Cronbach Alpha Subscale Reliabilities across Cycles 


Teacher Performance Assessed Fall 2016 Spring 2017 Fall 2017 
N=56 N=94 N=56 
0 oO oO 

Contextual Factors 5 items 5 items 5 items 

a=.907 a=.921 a=.673** 

4 items 4 items 4 items 
Learning Goals/Objectives a= .887 a=.850 a=870 

5 items 5 items 5 items 
Assessment Plan a=.903 a=.892 a=.821 

6 items 6 items 4 items 
Design for Instruction a=.924 a=882 a=.838 

3 items 3 items Omitted 
Instructional Decision Making a=.875 a=834 

4 items 4 items 4 items 
Analysis of Student Learning a=.882 a=897 Oa oe" 

5 items 5 items 6 items 
Reflection and Self-Evaluation a=.930 a=.875 a=.881 

3 items 3 items One item only 
Formatting Expectations* a=.683** a=.690** "References" 


*Not counted as a subscale. **Reliability below acceptable a=.800 

CAEP's value expectation for acceptable reliabilities is at least 800 (CAEP, 
2015). Cronbach's alpha measures reliability by the internal consistency of the items, 
that is, each of the items is measuring the same construct or content. Cronbach's alpha 
is used in the development and evaluation of reliability of instrument scales and 
subscales. 

Seven components were identified in the data through factor analysis. These 
subscales in each year of data are: Contextual Factors, Learning Goals/Objectives, 
Assessment Plan, Design for Instruction, Instructional Decision Making (omitted in 
Fall 2017), Analysis of Student Learning, and Reflection and Self-Evaluation. 
"Formatting Expectations" was not defined in this study as a subscale. 

The TPA data from the first two Cycles (Fall 2016, and Spring 2017) had seven 


identified subscales: Contextual Factors, Learning Goals/Objectives, Assessment Plan, 
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Design for Instruction, Instructional Decision Making, Analysis of Student Learning, 
and Reflection and Self-evaluation. The subscales were confirmed through factor 
analysis and each has a Cronbach's alpha internal consistency of .80 or higher. 
"Formatting Expectations" was not defined as a subscale in this study. 

The TPA data from the third Cycle (Fall 2017) resulted from a revised TPA for 
Fall 2017. Six subscales were confirmed in the Fall TPA data through factor analysis: 
Contextual Factors, Learning Goals/Objectives, Assessment Plan, Design for 
Instruction, Analysis of Student Learning, and Reflection and Self-evaluation. 

Reliability coefficients for two of the subscales in the Fall 2017 TPA data were 
below CAEP acceptable reliability levels: Contextual Factors (a=.673) and Analysis of 
Student Learning (a=.757). Results from the two subscales should NOT be used for 
making decisions about a candidate's knowledge and skills of in these two areas. Nor 
should the results from these two subscales be used in the aggregate. The lack of 
reliability in the subscales should be of concern to decision-makers. A review of data 
across Cycles should be helpful in making corrections to these "measures". 

Cronbach's alpha, as other measures of reliability, is a statistical calculation. 
The more variance in the data, the higher the measure of internal consistency (true of 
any correlational computation). For assessments, a "rule of thumb" is to have at least 
25 items with five points in the rating scale or four multiple-choice items for classroom 
assessments or research surveys. Thus, the more items used to measure a TPA scale, 
the higher the internal-consistency reliability. Alternatively, returning to the measures 
used in Cycles I and II should be helpful. 
Validity: Fairness and the Opportunity to Learn (OTL) the Requirements of the 
TPA 

Candidates are introduced to the Teacher Performance Assessment in the 
Teacher Education Handbook. Each year's handbook includes a copy of the TPA and 
Rubrics. Required courses have assignments aligned with the required sections of the 
TPA (see Appendix for course alignments with the TPA for three Cycles). The 
knowledge and skills are taught for mastery in the classes. Accommodations are 


offered in each class syllabus for candidates with documented learning differences. 
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Candidates have opportunities to demonstrate their learning within classrooms using 

feedback from university and clinical faculty. Candidates are given the opportunity 

throughout their work on the TPA to receive feedback and to re-do sections. 
Reliability 

CAEP Definition of Reliability and Inter-Rater Reliability (CAEP Glossary, 

2015): 

The degree to which test scores for a group of test takers are consistent over 
repeated applications of a measurement procedure and hence are inferred to be 
dependable and repeatable for an individual test taker. A measure is said to have a high 
reliability if it produces consistent results under consistent conditions. Tables 6 and 7 
below present Inter-Rater Agreement (Graham, Milanowski, & Miller, February 2012) 


for Cycles 1, 2, and 3. IRA provides exact scale agreement by at least two raters. 
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Table 6. Inter-Rater Agreement for Teacher Performance Evaluations: Cycles 1 and 2 


State or Local Standards 


Standards CYCLE 1: Fall 2016 N=56 CYCLE 2: Spring 2017. N=94 
Contextual Factors IRA %* % Rated | MEAN | MODE | IRA %* /|% Rated 3 0r) MEAN | MODE 
3 or 4** 4** 
Knowledge of 81.1% 98.1% 3.07 3.00 73.7% 95.0% 3.60 4.00 
Community, School and 
Classroom Factors 
Knowledge of 69.8% 94.3% 303 3.00 63.1% 93.0% a1 4.00 
Characteristics of Students 
Knowledge of Students’ 86.7% 88.7% 3.24 3.00 57.4% 81.9% Sf 4.00 
Varied Approaches to 
Learning 
Knowledge of Students’ 84.9% 86.8% | 3.00 62.7% 81.9% 3.40 4.00 
Skills and Prior Learning 
Implications for 90.6% 84.90% 3.16 3.00 70.2% 86.7% 339 4.00 
Instructional Planning and 
[Assessment 
- IRA % % Rated3| MEAN | MODE, IRA %* |% Rated 3 0r)| MEAN | MODE 
Learning Goals or 48 ree 
Significance, Challenge 83.0% 88.7% 3.31 4.00 74.4% 95.2% 3.68 4.00 
and Variety 
Clarity 90.6% 98.1% 3.47 4.00 77.6% 95.2% 3.70 4.00 
Appropriateness for 88.7% 98.1% 3.38 3.00 76.6% 94.8% 3.65 4.00 
Students 
Alignment with National, 96.2% 96.2% 3.50 4.00 77.71% 94.8% 3.70 4.00 


*IRA=Percent of TPAs with Rater Exact Score Agreement (Scale range 1-4). **Percent of TPAs rated 3 or 4 (Adjacent Score). 
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Table 6. Inter-Rater Agreement for Teacher Performance Evaluations: Cycles 1 and 2 (Continued) 


Standards CYCLE 1: Fall 2016 N=56 CYCLE 2: Spring 2017. N=94 
Assessment Plan IRA %* | % Rated 3| MEAN MODE | IRA %* | % Rated3| MEAN MODE 
or 4** or 4** 
Alignment with Learning Goals 88.7% 92.55 3.33 3.00 64.9% 93.3% 3.49 4.00 
and Instruction 
Clarity of Criteria and Standards 86.8% 92.4% 3.34 4.00 71.3% 93.3% 3.54 4.00 
for Performance 
Multiple Modes and Approaches 88.7% 92.4% 3.36 4.00 55.3% 92.4% 3.40 4.00 
Technical Soundness 88.7% 92.4% a2} 3.00 NOT ASSESSED 
Adaptations Based on the 88.7% 81.1% 3.24 3.00 70.2% 84.9% 3.35 3.00 


Individual Needs of Students 


IRA % |% Rated3| MEAN MODE | IRA % * | % Rated3| MEAN MODE 


Design for Instruction or 48" or 4" 
Alignment with Learning Goals 88.7% 94.33% 3.37 4.00 63.5% 96.7% 3.67 4.00 
Accurate Representation of 88.7% 94.33% eeu 3.00 63.5% 95.7% 3.48 4.00 
Content 
Lesson and Unit Structure 96.2% 92.4% 3.35 3.00 73.4% 95.7% 3:52 4.00 
Use of a Variety of Instruction, 88.7% 86.8% 5229 3.00 68.0% 88.7% 3.50 4.00 
Activities, Assignments and 
Resources 
Use of Contextual Information 88.7% 88.7% 321 3.00 68.0% 88.7% 3.54 4.00 


and Data to Select Appropriate 
and Relevant Activities.. 


Use of Technology 86.8% 92.4% 3.47 4.00 72.3% 90.5% 3.56 4.00 
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Table 6. Inter-Rater Agreement for Teacher Performance Evaluations: Cycles 1 and 2 (Continued) 


Standards CYCLE 1: Fall 2016 N=53 CYCLE 2: Spring 2017 N=94 
Instructional Decision-Making | IRA %* |Rated3 or4 | MEAN | MODE IRA %* Rated 30r4 | MEAN | MODE 
Sound Professional Practice 92.5% 92.4% 3.41 4.00 79.7% 94.6% 3.66 4.00 
Modifications Based on Analysis 100% 92.5% 3.33 3.00 63.8% 92.4% 3.45 4.00 
of Student Learning 
Congruence Between 92.5% 88.7% 3.26 3.00 61.8% 91.4% 3.32 3.00 
Modifications and Learning Goals 
Analysis of Student Learning | IRA % (|Rated3o0r4 | MEAN | MODE | IRA % * | Rated3 or4 | MEAN | MODE 

Clarity and accuracy of 75.5% 90.6% 3.34 3.00 67.0% 89.2% 3.56 4.00 
Presentation 
Alignment with Learning Goals 84.9% 86.8% 3.31 3.00 67.0% 88.2% 3.54 4.00 
Interpretation of Data 86.8% 84.9% 3.17 3.00 54.2% 84.9% 3.38 4.00 
Evidence of Impact on Student 86.8% 88.7% 3.20 3.00 62.7% 91.4% 3.38 4.00 
Learning 

Reflection and Self-Evaluation IRA %_ | Rated3 or4 | MEAN | MODE | IRA % * | Rated3 or4| MEAN | MODE 
Interpretation of Student Learning} 88.7% 86.8% 3.20 3.00 61.7% 89.2% 3.37 4.00 
Insights on Effective Instruction 86.8% 88.7% 3:13 3.00 58.5% 84.9% 3.41 4.00 
and Assessment 

Alignment Among Goals, 84.9% 83.0% 3.20 3.00 63.8% 89.2% 237 4.00 
Instruction and Assessment 


*TRA=Percent of TPAs with Rater Exact Score Agreement (Scale range 1-4). 


**Percent of TPAs rated 3 or 4 (Adjacent Score). 
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Table 6. Inter-Rater Agreement for Teacher Performance Evaluations: Cycles 1 and 2 (Continued) 


CYCLE 1: Fall 2016 N=56 


CYCLE 2: Spring 2017 N=94 


Reflection and Self-Evaluation | IRA % |Rated30r4) MEAN | MODE | IRA % * | Rated3 or4| MEAN | MODE 
Implications for Future Teaching 88.7% 90.5% ems, 3.00 53.290 89.2% 3.39 4.00 
Implications for Professional 79.2% 90.5 3.10 3.00 65.9% 91.4% 3.29 3.00 
Development 

Format Expectations IRA % _ |Rated 3 0or4) MEAN MODE | IRA % * | Rated3 or4| MEAN | MODE 
Format and Organization 94.3% 98.1% 3.85 4.00 73.4% 90.3% 3.69 4.00 
Writing 86.8% 92.4% 3.76 4.00 62.7% 93.8% 3.44 4.00 
References 98.1% 98.1% 3,94 4.00 77.6% 96.75 3.77 4.00 


*TRA=Percent of TPAs with Rater Exact Score Agreement (Scale range 1-4). **Percent of TPAs rated 3 or 4 (Adjacent Score). 
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Table 7. Teacher Performance Assessment (TPA) Evaluations: Cycle 3 Fall 2017* 


IResources 


Standards CYCLE 3: Fall 2017 N=56 
Contextual Factors IRA %* % Rated 3 or 4 | MEAN MODE 
Ee Pee or ean 3.2% 94.7% 3.78 4.00 
ommunity 
The Setting for learning - School, 
Classroom and Students cee he at ae 
The Setting for Learning 73.2% 98.2% 3.78 4.00 
The Setting for Learning Selection 
of Monitored Students Oe Se ot — 
The Setting for Learning Impact of 64.2% 85.7% 361 4.00 
Contextual Factors 
Learning Objectives IRA % * | % Rated3 or 4 | MEAN MODE 
Planning Sequence Organization 66.0% 16.8% 3.25 4.00 
Planning Sequence Objectives 71.4% 75.0% 3.48 4.00 
Planning for Content Alignment 4.00 
(National/State Standards) ee ee at 
Planning Academic Language 58.9% 76.7% 3.49 4.00 
Assessment Plan IRA % *_ | % Rated3 or 4 | MEAN MODE 
Planning Pre-Assessments 58.9% 69.6% 3.41 4.00 
Planning Multiple Assessment 
Modes for Formative Assessment.. sas ae aa oe 
Planning Classroom Assessment 
with Student Learning 35.7% 64.2% 3:23 4.00 
Planning Classroom Assessment 
IMeunesefor Dedenere 39.3% 64.3% 3.11 3.00 
Planning Classroom Assessment 
Based on Target Students Needs seule ee at saad 
Design for Instruction IRA % * | % Rated3 or 4 | MEAN MODE 
Alignment of Learning Objectives 60.7% 26.8% 3.61 4.00 
Engaging the Learner 57.1% 83.9% 3.45 4.00 
Measuring Learning Growth of the 46.1% 87.5% 
rae 3.49 4.00 
Use of a Variety of Instruction, 
Activities, Assignments and 41.1% 78.5% 3.34 4.00 


NOTE: The TPA was revised prior to the Fall 2017 Assessment. 


*IRA=Percent of TPAs with Rater Exact Score Agreement (Scale range 1-4). 


**Percent of TPAs rated 3 or 4 (Adjacent Score). 
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Table 7. Teacher Performance Assessment (TPA) Evaluations: Cycle 3 Fall 2017* 


(Continued) 


Standards 


Cycle 3: Fall 2017 N=56 


Design for Instruction 


; IRA % * |% Rated 3 or 4| MEAN MODE 
(continued) 
Modifications Based on Learners 41.1% 78.5% 3.33 3.00 
Needs 
* 

Analysis of Student Learning Lee HOPES SOG NILE IN ee 
Clarity and accuracy of Whole 62.5% 98.2% 
Group Learning ou a 
Impact of Whole Class Learning 42.8% 82.1% 
Feedback 3.40 4.00 
Interpretation of Data of Target 41.4% 82.1% 
ieremeee 3.35 3.00 
Evidence of Impact of Target 50.0% 75.0% 
Student 3.28 3.00 

2 

Reflection and Self-Evaluation ee ie aled oid eee ical 
Interpretation of Student Learning 46.4% 67.8% 3.20 3.00 
‘Teacher Candidate Insight on 41.0% 75.0% 
Effective Instruction and FO) 3.00 
Assessment 
Self-evaluation and Reflection to 46.40% 76.6% 3.24 3.00 
Improve Planning and Practice 
Implications for Future Teaching 44.6% 60.7% 3.09 3.00 
Format and Organization 71.4% 94.6 3.64 4.00 
Writing 42.8% 89.2% 3.39 3.00 
References 82.1% 94.6% 3.86 4.00 


NOTE: The TPA was revised prior to the Fall 2017 Assessment. 


*IRA=Percent of TPAs with Rater Exact Score Agreement (Scale range 1-4). 


**Percent of TPAs rated 3 or 4 (Adjacent Score). 
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Interpretation of Reliability Tables for TPA Cycles 1, 2, and 3 

The Tables present data for three cycles of the Teacher Performance Assessment (TPA). 
Cycle 1 is Fall 2016; Cycle 2 is Spring 2017; Cycle 3 is the "revised" TPA data from Fall 2017. 

The data were analyzed using Excel. Note the variance in the IRA (Inter-Rater 
Agreement) by TPA task (Graham, Milanowski, & Miller, February, 2012). The IRA is exact 
score agreement, a measure that is now recommended for evaluations in education---especially 
in the ratings of teacher performance. The IRA replaces IRR--inter-rater reliability. IRR was 
often confusing, and calculation was dependent on the number of points on the rating scale used. 
IRA is a more transparent measure. 

The IRA percentage is interpreted by the percentage of TPAs that received exactly the 
same rating. This is presented by individual "tasks" of TPA in order to provide instructional 
feedback for professors and candidates. 

Note the IRA for selecting the two highest scores on the scale--indicate the percentage of 
TPA that were scored either a 3 or 4. Please note that the reason that the IRA (exact agreement) 
las lower percentages is that the majority of the "disagreement" in rating is apparently between 
the scores of 3 or 4.Care should be taken in future TPA scorer training sessions to spend enough 


time on helping scorers make clear distinctions between the ratings of "3" and "4." 
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Validity: Fairness and the Opportunity to Learn (OTL) Knowledge and Skills for the TPA 
MSU EPSB Key Assessments / CAEP 3-Cycle Assessments--Initial Certification 
Programs--Cycle/Course identification--08/ 2017 draft 


EPSB area #4 | EPSB area #5 | EPSB area #6 | EPSB area #7 | EPSB area #8 | EPSB area #9 
Observation | Assessment Contextual Lesson Plan Literacy Professional 
Plan Factors Assessment | Attributes and 
Dispositions 
(Phil. Paper, 
Ethics, etc.) 
Program |Cycl|Cyc2|\Cyc3|Cyel|Cye2|Cye3|Cyc1)Cyc2|Cyc3\Cycl|\Cyc2|\Cye3|\Cycl1|Cyc2|\Cyc3|Cyel|Cye2|Cyc3 
(Cour|Cour|Cour|Cour/Cour|Cour|Cour|Cour/Cour/Cour|Cour/Cour|Cour|Cour|Cour/Cour|Cour|Cour 
se | se | se | se | se | se | se | se | se | se | se | se | se | se | se | se | se | se 
TECE IECEIECEIECEIECE] EDS [ECE] EDE [ECEIECE|EDE |IECEIECE|EDE]EDS [ECE] EDF] TEP ECE 
361 | 418 | 425/255) P | 425] C |301|425| E |345/425| E | P | 425 | 207) AD | 425 
230 253 305 327 | 320 M 
IP-5 EDE|EDE|TPA|EDS|EDS|TPA|EDF|EDS|TPA/EDE| SCI |TPA|EDE|EDE|TPA|EDF| TEP |TPA 
M|E iP | IP Dili | IP E | 490 E|E 207 | AD 
330 | 321 230 | 367 365 305 |EDE 327 | 331 M 
E 
322 
IP-5/ LBD P- |EDE|EDS | TPA|EDS |EDS |TPA|EDF/EDS|TPA|EDE] SCI | TPA|EDE|EDE]|TPA|EDF|TEP |TPA 
12 M | P iP || ip Dili || IP E | 490 E|E 207 | AD 
330 | 359 230 | 367 365 305 |EDE 327 | 331 M 
E 
322 
IP-5 /MSD |EDE|EDS|TPA|EDS|EDS|TPA|EDF/EDS|TPA|EDE] SCI |TPA|EDE|EDE|TPA|EDF| TEP |TPA 
IP-12 M | P P | Pp 211} P E | 490 iB || 1B 207 | AD 
330 | 375 230 | 367 365 305 |EDE 327 | 331 M 
E 
322) 
MIDDLE |/EDE|EDM|TPA|EDS |EDU|TPA|EDF|MET|TPA |EDM|MET]| TPA |EDMIEDM| TPA |EDF| TEP |TPA 
M|G P | ¢C 211) H G|H G/|G 207 | AD 
330 | 332 230 | 482 306 347 | 332 M 
MIDDLE / |EDE|EDS|TPA|EDS|EDU|TPA|EDF/EDS|TPA|EDM|EDS | TPA IEDMIEDM EDF! TEP |TPA 
ILBD P-12 M | P P | C Dili] | 1 G || G/|G 207 | AD 
330 | 359 230 | 482 365 306 | 357 347 | 332 M 
IMIDDLE/ |EDE|EDS|TPA|EDS|EDS |TPA|EDF|/EDS|TPA |EDM|MET] TPA |EDMIEDM| TPA |EDF| TEP |TPA 
IMSD P-12 | M | P RP | Pp Dili || IP G|H G/|G 207 | AD 
330 | 375 230 | 375 365 306 347 | 332 M 
EDU 
C 
482 
SOCIAL EDS|EDS|EDS|EDS |EDF|EDS|EDF|EDS|EDS|EDS|EDS|EDS| EDUC 476 |EDF/TEP/TPA 
STUDIES 8-| E | E | E |} P /311|} E |]211| E | E| E}] E/E 207 | AD 
12 451?|451?| 416 | 230 416 451?| 416 |451?/451?| 416 M 
EDS|EDS EDS EDS|EDS 
E/E E E/E 
499D499D 499D| 499D499D| 
2 2 ? 2 ? 
IPHYSICS 8-) SCI | SCI |EDS|EDS |EDF|EDS|EDF| SCI |EDS| SCI | SCI}EDS| EDUC 476 |EDF/|TEP|TPA 
12 402 |402) E | P |311] E |211/402) E |402)402) E 207 | AD 
416 | 230 416 416 416 M 
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SCI | SCI SCI SCI | SCI 
403 | 403 403 403 | 403 
(CHEMISTR| SCI | SCI |EDS|EDS|EDF|EDS|EDF| SCI |EDS| SCI} SCI)/EDS| EDUC 476 |EDF|TEP|TPA 
YY 8-12 402 | 402) E |} P {311} E | 211/402) E |402)402) E 207 | AD 
416 | 230 416 SCI | 416 | SCI | SCI | 416 M 
403 403 | 403 
BIOLOGY | SCI | SCI |EDS|/EDS|EDF/EDS|EDF) ? |EDS| ? 2? |EDS| EDUC 476 |EDF|TEP|TPA 
8-12 402 |402) E | P {311} E /} 211 E E 207 | AD 
416 | 230 416 416 416 M 
IMATH 8-12) SCI | SCI]EDS|EDS|EDF|EDS|EDF| ? |EDS| ? ? |EDS| EDUC 476 |EDF|TEP|TPA 
402 |402) E | P {311} E / 211 E iE) 207 | AD 
416 | 230 416 416 416 M 


SPANISH ? ? |EDS|EDS|EDF|EDS|EDF| ? |EDS| ? 2? |EDS| EDUC 476 |EDF|TEP|TPA 


IP-12 1 | IP | sili | 1s, |] 2il E E 207 | AD 
416 | 230 416 416 416 M 
IMUSIC P- |MUSIMUS|EDS|EDS |MUS|EDS|EDF|MUS|EDS|MUS/MUS|EDS| EDUC 476 |EDF|TEP|TPA 
12 B | 2 | i | PP | i | is ail is | ie | ie | i | 207 | AD 
215 | 207 | 416 | 230 |375/| 416 375/| 416 | 207 |375/| 416 M 
376 376 376 
IMUS 
E 
B25) 
ART P-12 JART|ART|EDS|EDS| ? |EDS|/EDF/ART|/EDS|ART|ART|EDS| EDUC476 |EDF|TEP|TPA 
300)301; E | P E | 211)300} E | 300)301/ E 207 | AD 
ART] 416 | 230 416 416 ART] 416 M 
321 321 


THEATRE |THE|THE|EDS|EDS|EDF/EDS|EDF/THE|]EDS|THE|THE/EDS| EDUC476 |EDF|TEP|TPA 


IP-12 A || A | Je | I | Sli} de jazi | A | Js |] A |] AY | Js 207 | AD 
375 |475?| 416 | 230 416 375 | 416 | 375 | 475 | 416 M 
THE THE THE 
A A A 
370 370 370 
THE 
A 
475 
IENNGLISH |/ENG|ENG|EDS|EDS|EDF|EDS|EDF/ENG|EDS|ENG|ENG|EDS| EDUC 476 OR |EDF| TEP |TPA 
3-12 382/400} E |} P |311) E |211)382) E | 382|400} E |ENG 381, ENG| 207 | AD 
416 | 230 416 416 416 | 382, and EDSE M 
416 
IBUSINESS | BIS | BIS |EDS|EDS|EDF/EDS|EDF) BIS |EDS| BIS | BIS |EDS| EDUC476 |EDF|/TEP|TPA 
INFO 499C499C) E | P | 311] E | 211 499C) E 499C/499C) E 207 | AD 
(TECH 8-12 416 | 230 416 416 416 M 


